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1. Introduction 



Abstract. A range of nonlinear image reconstruction procedures 
based on extremizing the generalized Shannon entropy, Kullback- 
Leibler cross-entropy and Renyi information measures and proposed 
by the author in early papers is presented. The "generalization" as- 
, sumes search for the solution over the space of real bipolar or complex 

■^j- ■ functions. Such an approach allows, first, to reconstruct signals of any 

O | type and physical nature and, secondly, to decrease nonlinear intensity 

CN . image distortions caused by measurement errors. All the elaborated 

■ procedures are contained in VLBI "IMAGE" program package devel- 

; oped in IAA RAS. 

6 
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The main aim of this paper is consideration of a class of image deconvolution (re- 
^ ■ construction) algorithms based on optimization of different information measures. The 

most popular among them is the well known maximum entropy method in Shannon for- 
mulation. The generalized information methods based on extremizing entropic (Shan- 
non, Kullback-Leibler) and Renyi measures were proposed by the author in several 
previous publications [1-3] for reconstruction of real bipolar and complex images. In 
this paper we give a brief review of the proposed techniques. 

All the proposed methods are realized in "IMAGE" program package elaborated 
in IAA RAS in framework of " QUASAR" Russian Very Long Baseline Interferometry 
(VLBI) project. 



2. Entropic Processing 



2.1. Maximum Entropy Method 



Maximum entropy method (MEM) is widely used for image reconstruction from 
incomplete and noisy Fourier data. In Fourier space the image reconstruction problem 
is equivalent to one of extrapolation and interpolation. The MEM produces maximally 
smooth, or unbiased, estimates [4]. If r m i is a real non- negative signal (we consider 



two-dimensional sampling signals) to be estimated then the maximum entropy method 
assumes solving the following optimization problem with linear constraints dictated by 
data 

max]T£-(r m ;m(r m 0) - 1/2 + r ml > 0, (1) 

ml n k 

E E T mA + Vnk = Kk, E E r mlV% + Vnk = B nk , (2) 
ml ml 

where a^,6J^ are constants determined by linear image formation system (Fourier 
transform), A nk) B nk are the real and imaginary parts of the Fourier data respectively, 
rfnkirfnk are noise components of data, o\ k are noise variances for each of the real and 
imaginary parts at sample nk. Note that the signal must be real and non-negative- 
definite for a real entropy functional to exist. 

Using the Lagrange method it is easy to obtain a solution for r m i and noise compo- 
nents 

r m i = exp(- E E «^ fl mi + Pnkb^i - 1), (3) 

n k 

Vnk = °lk a nk, ink = a lkPnk, (4) 

where a n k, /3 n k are conventional Lagrange multipliers. 

2.2.Kullback-Leibler Entropy Method 

Kullback-Leibler information permits insertion of a prior bias function a m i into en- 
tropy functional (1) as follows 

EE r ^ ln ( r ^/°™') ? r mh a mt >0. (5) 

m l 

The aim of such insertion is improving the entropic algorithm (1) using a priori informa- 
tion about an unknown signal [2]. Usually as a bias function a m i a first approximation 
to r m i obtained by inverse Fourier transform of input Fourier data is used. Missing data 
regions in Fourier space are filled out by zeros. 

3. Renyi Information Processing 

Another powerful method of reconstruction considered here is based on minimization 
of the following Renyi information measure [3] 

EE<Xn7, r ml ,a m i>0, a^O, (6) 

m l 

where a m i is a reference function. 

4. Generalized Methods 

Originally the generalized maximum entropy method (GMEM) was proposed by 
Bajkova [1] for reconstruction of complex coherent images formed by radio holography 



principle. In [2] the GMEM idea was used by Frieden for generalization of Kullback- 
Leibler cross-entropy reconstruction of ISAR images. In [3] by Frieden the general- 
ized form of entropy functionals was extended to minimum Renyi information method. 
Bajkova proposed the method of solution of corresponding generalized optimization 
problem [3]. 

4.1. Generalized Maximum Entropy Method 

Here let us recall the basic aspects of the GMEM [1]. 

First, consider a real signal r m i which may take both positive and negative values. In 
this case the minimizing functional cannot be written as (1). Therefore it was proposed 
in [1] to represent entropy (1) as entropy of the absolute value of r mi : 

EE Mhi(M). (7) 

m I 

To represent the optimization problem in traditional form (l)-(2) let us represent 
the signal sought for r m \ as a difference between two non-negative-definite functions: 

r m \ = Xrnl - Vmh %ml > 0, y m l > 0. (8) 

Thus the positive regions are represented by x m i and the negative by y m i. Curves 
x m i and y m i do not have overlapping support regions. Nonoverlapping condition can be 
written as 

if r ml > then y ml -> and r ml = x mh 
if r mi < then x mi -> and r m i = -y m i- 

Then the problem of estimation of r m i can be replaced by one of estimation of x m \ 
and y„ d as 

maxE E ~( X ml H X ml) + yml HVml)) ~ 1/2 E E«fc + Vnk) / °lk- ( 9 ) 
m I n k 

Jumping to a complex signal u ml = r rnl + jq ml , j = ^1, where both r ml and q ml 
are real bipolar signals, we can represent similarly to (8) the sought for sequences as 
the differences of non-negative-definite sequences as well: 

f'ml = Xmi — Vmh Qml = Z-ml ~ "ml, (10) 

where x mi > 0, y„ d > 0, z„ d > 0, v ml > 0. 

Obviously, if the sequences x m i,y m i and z m i,v m i do not overlapp, then x m i and z m \ 
determine positive parts and y m \ and v m i determine negative parts of the sequences r m i 
and q m i respectively. 

Nonoverlapping condition mentioned above must be complemented by one for the 
signal q ml : 



if q m i > then v mi -> and q mt = z mU 
if q„ d < then z„ d -> and q ml = -v„ d . 



Then the minimized entropic functional for estimation of the complex signal can be 
written as the following functional 

E E X ™' M ax ml) + Vml M a Vml) + z ml M az ml) + V ml M av ml) ( U ) 
m I 

with inserted a positive parameter a responsible for not overlapping positive and neg- 
ative parts of the search for real bipolar sequences r m i and q m \- Significance of the 
parameter a will be clear below from equations (15)-(17). 

Linear constraints (2) derived from Fourier data must be rewritten in the way: 

E E ( X ml ~ VraMtl ~ ( Z ml ~ V m l)b^ + rf nk = A nk , (12) 
m I 

E E ( X ™1 - yml)V% + (Zml ~ V m l)a^ + lf nk = B nk , (13) 
m I 

Xml > 0, Vml > 0, Z m i > 0, V mt > 0, (14) 

where a^^b^ are constant coefficients which are determined by Fourier transform, 
A nk , B nk are measured real and imaginary parts of Fourier data respectively. The 
solutions of optimization problem (11)-(14) obtained using the Lagrange method for 
signals (x mh y mh z mh v ml ) look like 

x ml = exp(- E E a nk0^i + PnkU^i - 1 - In a), (15) 

n k 

v ml = exp(- E E - Pnka^ - 1 - In a), (16) 

n k 

XmiVmi = ZmiVmi = exp(-2 - 2 In a). (17) 

Making parameter a larger we can reach not overlapping effect. Usual value of a is 
1000. 

Unknown Lagrange multipliers can be found from the following dual optimization 
problem without supplementary conditions 

maxE E ~( X ml + Vml + Z m l + V m l) - 1/2 E E^nfc + Vnk)/°nk 
ml n k 

~EE ( a nkA nk + (3 nk B nk ) "EE ( a nkVnk + PnkVnk) ( 18 ) 
n k n k 

by substituting expressions (15)-(17) for x mh y mh z mh v m i and (4) for rf nk ,rf nk . 

Equations (15)-(17) show that for any real solution {a nk }, {j3 nk } to the problem, 
necessarily 

Xml > 0, Vml > 0, Z m i > 0, V m l > 0. (19) 

That is all the reconstructed signals are non-negative. This is important because non- 
negative-constrained solutions, being nonlinear in the data, can have higher bandwidth 
and hence higher resolution than the data. By (10) this is now true for real signals r m i 
and q m i and hence for complex signal u m i as well. 

So, the GMEM approach allows to obtain a solution of the reconstruction problem 
in the space of complex functions. Because of non-negativity of the solutions (15)-(17) 



the method is nonlinear and possesses a super resolution effect similarly to the classical 
maximum entropy method. 



4. 2. Generalized Kullback-Leibler Method 



Generalized Kullback-Leibler method assumes prior knowledge of bias non-negative 
functions a mh b mh c m i, d m i corresponding to the signals x mh y mh z mh v m i respectively 
Then the entropic functional (11) is modified as 

^ZjX m i \n(ax m i/a„ d ) + y ml \n(ay m i/b ml ) 

m I 

+z m[ ln(az m i/c m i) + v m i \n(av m i/d m i)- (20) 

In this case, bias functions can be again the smoothed inverse Fourier transforms of 
Fourier data with zeros in the points where measurements are absent. 

Using data constraints (12)-(13) and the Lagrange method we obtain the following 
solution to sought for non- negatively determined signals 



x m i = (a m i/ea) exp(- J2 J2 oc nk a^ + (3 nk b^ - 1), (21) 

n k 

Vmi = {d m i/ea) exp(- ]T ]T a nk b^ - /3 nfc a^ - 1), (22) 



n k 

-2„ -2 



XmiVmi = a m ib m ie a , (23) 

^ml^ml Cmldml& ^ • (24) 

As seen, maximizing Kullback-Leibler entropy functional subject to the linear data 
constraints (12)-(13) produces solutions (21)-(24) that are linearly proportional to the 
input functions a mh b mh c mh d mX respectively. 



4. 3. Generalized Minimum Renyi Information Method 



Similarly to maximum entropy method for complex signal u m \ = r m i + jq m i we 
suggest [3] to minimize the sum of two Renyi measures of absolute values of r m i and 
q m \. Then functional (6) is modified as 

EE-(Kr^7 + k m r^7), (25) 

m I 

where a m i and b m \ are input modulus reference signals. 

Representing the sought for sequences in the form of differences between positive 
and negative parts as in (10) we can rewrite previous functional (25) as follows 

X X ~( X ml a lni a + ymfimi + ^m^mi" + ^mi^mz")) (26) 
m I 

where a m i,b m i,c m i,d m i are corresponding reference signals. 

Direct using the Lagrange method is very complicated. In order to simplify solution 
let us make the following substitution a — 1 + 1/2/x and introduce the following new 
variables: 

tml{ s mli hml> 9ml) = {x m l{ s ml7 h m i, 9ml) I ^mlipmh c mh d m l)) ^ ^ ■ (27) 



Then the optimization problem (20), (12)-(14) can be rewritten with respect to new 
variables (27) as 

max E E -( a mit 2 S +1 + b m is 2 Z +1 + c m ih 2 Z +1 + d ml g%i +1 ) 

m I 

-1/2 EEfaS + (28) 

n k 

E E ( a mitla - bmis^a^ - (cmih^i - dmig^b^ + rf nk = A nk , (29) 

m I 

E E (Am* - bmls'X + (Crthfo ~ d^g^d* + Vnk = B n k, (30) 
m I 

tml > 0, S m l > 0, h m l > 0, gW > 0. (31) 

Using the Lagrange method we find the following solution for sought for signals: 

t ml = -2/x/(2/i + 1) E E + (32) 

n k 

h ml = 2/i/(2/i + 1) E E ^nkbti - Pnka^, (33) 

" k 

tml = Sml', hml = —Qml- (34) 

where a n k and f3 n k are Lagrange multipliers. 

Remembering condition (31) and taking (34) we can say that the negative and 
positive parts of sought for real solutions to r m \ and q m i never overlapp and no parameter 
(as a in the GMEM) is required. That is an advantage of the generalized minimum 
Renyi information method. 

Sought for a complex signal u rn i can be found by inverting equations (27) and using 
representation (10). 

It is necessary to note that the quality of image reconstruction by Renyi method 
depends on choosing parameter a in the functional. When a = 2(/i = 0.5) we have 
a conventional minimum intensity criterion. Experimentally established that when a 
approaches 1 by increasing \i we obtain a criterion with more strong nonlinear (extrap- 
olation) features [5]. 



5. A problem of nonlinear distortions 

Let us consider real non-negative images. Because of nonlinearity nature of con- 
sidered above deconvolution procedures there is a problem connected with nonlinear 
image distortions due to noise in spectral data [6]. We investigated only the case of 
additive noise in data. As experiments show the generalized algorithms ensure much 
less artefacts than the classical ones. This phenomenon can be explained that noise 
can so degrade data that to the latter in general a real solution with both positive and 
negative values will correspond but not pure real non-negative one which is sought by 
conventional algorithms. Therefore we consider that seeking for a solution of image 
reconstruction problem in generalized form is more adequate. 



6. The "IMAGE" program package 



"IMAGE" program package elaborated in IAA RAS [7] in framework of "QUASAR" 



VLBI project is intended for VLBI mapping natural (incoherent) and artificial (coher- 
ent, ISAR) radio sources and investigation of new image reconstruction procedures. 
At present the "IMAGE" program package contains except of the CLEAN and clas- 
sical information nonlinear methods a number of generalized versions of information 
methods. For solution of related optimization problems both steepest descent and 
Newton-Raphson numerical methods are realized. 

7. Conclusion 

Maximum entropy, maximum Kullback-Leibler cross-entropy and minimum Renyi 
information methods having good extrapolation and interpolation features are gener- 
alized for reconstruction of signals of any type including complex ones. They may be 
used for reconstruction of real bipolar and complex signals or reconstruction of real 
non-negative images with minimal nonlinear distortions. 

This work was supported by Russian Foundation for Basic Researches under grant 
N 96-02-19177. 
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